Support UDTFs in information_schema.routines / SHOW FUNCTIONS#23438
Open
zhuqi-lucas wants to merge 1 commit into
Open
Support UDTFs in information_schema.routines / SHOW FUNCTIONS#23438zhuqi-lucas wants to merge 1 commit into
zhuqi-lucas wants to merge 1 commit into
Conversation
github-actions
Bot
added
sql
zhuqi-lucas
force-pushed
the
qizhu/show-functions-udtf-upstream
branch
from
f297568 to
55cb62d
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends DataFusion’s SQL discovery surfaces to include table functions (UDTFs) in information_schema.routines and SHOW FUNCTIONS, addressing the gap where only scalar/aggregate/window UDFs were listed.
Changes:
- Snapshot
SessionState.table_functionsintoInformationSchemaProviderso UDTFs can be exposed without introducing crate cycles. - Emit
information_schema.routinesandinformation_schema.parametersrows for UDTFs (syntheticOUTparameter row withdata_type = 'TABLE'). - Rewrite
SHOW FUNCTIONSSQL to start fromOUTparameter rows andLEFT JOININparameter rows so functions with no inputs can appear.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
datafusion/sql/src/statement.rs |
Rewrites SHOW FUNCTIONS to be compatible with functions that lack IN parameters (e.g., UDTFs). |
datafusion/core/src/execution/session_state.rs |
Passes the session’s registered table functions into the per-query InformationSchemaProvider. |
datafusion/catalog/src/information_schema.rs |
Stores table functions in InformationSchemaConfig and emits corresponding routines / parameters rows. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+2646
to
2649
| o.specific_name function_name, | ||
| o.data_type return_type, | ||
| array_agg(i.parameter_name ORDER BY i.ordinal_position ASC) parameters, | ||
| array_agg(i.data_type ORDER BY i.ordinal_position ASC) parameter_types |
Comment on lines
+320
to
+322
| // Table functions (UDTFs) don't have scalar signatures; their return | ||
| // type is always a table, so emit a single row per UDTF with | ||
| // routine_type = "TABLE" and data_type = "TABLE". |
Comment on lines
2634
to
2636
| // Note: use LEFT JOIN from OUT rows to IN rows so functions without | ||
| // input parameters (notably UDTFs / table functions) still appear. | ||
| let query = format!( |
zhuqi-lucas
force-pushed
the
qizhu/show-functions-udtf-upstream
branch
from
55cb62d to
3ede8a3
Compare
zhuqi-lucas
force-pushed
the
qizhu/show-functions-udtf-upstream
branch
from
3ede8a3 to
129e008
Compare
- add table_functions field to InformationSchemaConfig + builder - emit UDTFs from make_routines (function_type=TABLE, data_type=TABLE) - emit synthetic OUT parameter for UDTFs so JOIN in SHOW FUNCTIONS resolves - rewrite SHOW FUNCTIONS SQL as OUT LEFT JOIN IN so args-less UDTFs surface - wire SessionState.table_functions snapshot into InformationSchemaProvider
zhuqi-lucas
force-pushed
the
qizhu/show-functions-udtf-upstream
branch
from
129e008 to
3fb1e9a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #23437.
Rationale for this change
information_schema.routinesandSHOW FUNCTIONScurrently enumerate only scalar / aggregate / window UDFs. Table functions (UDTFs) registered viaSessionContext::register_udtfare omitted, making them undiscoverable through SQL. Discovery matters because downstream tooling (DataFusion CLI, Massive's atlas SQL surface, dbt-datafusion, etc.) uses these SQL surfaces to list available functions.What changes are included in this PR?
1. Snapshot table functions into the information_schema provider.
TaskContextcannot expose table_functions without introducing a crate cycle (TableFunctionlives indatafusion-catalog, which depends ondatafusion-executionwhereTaskContextlives). Instead, snapshotSessionState.table_functionsintoInformationSchemaProviderat construction time. The provider is built per-query viaschema_for_ref, so the snapshot stays fresh.2. Emit UDTF rows.
make_routines: one row per UDTF withroutine_type = \"FUNCTION\",function_type = \"TABLE\",data_type = \"TABLE\".make_parameters: a single syntheticOUTrow per UDTF (data_type = \"TABLE\") so the JOIN inSHOW FUNCTIONSresolves. UDTFs have no scalar signature, so noINrows are emitted.3. Rewrite the SHOW FUNCTIONS SQL.
The old query joined
parameters p (INNER)requiring both IN and OUT rows before joiningroutines. UDTFs have no IN parameters, so they fell out. New shape:```
FROM (parameters WHERE mode='OUT') o
LEFT JOIN (parameters WHERE mode='IN') i
ON matching keys
JOIN information_schema.routines r
ON name matches
```
Are these changes tested?
Verified locally by registering a UDTF and running SHOW FUNCTIONS:
```
Happy to add sqllogictests in
datafusion/sqllogictest/test_files/information_schema.sltif maintainers prefer.Are there any user-facing changes?
Yes: SHOW FUNCTIONS output now includes table functions.
information_schema.routinesandinformation_schema.parametersgain rows for each registered UDTF. Existing rows are unchanged.